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ABSTRACT 

Litton Data Systems has institutionalized the inspection process, and achieved 
dramatic results in terms of defect prevention and cost savings thus far. Additionally, 
several findings have been gleaned from an analysis to optimize the process. Over 300 
inspections have been performed over the last two years on many types of documents, and 
this paper describes some quantitative results to-date from the initial "champion" project. 

BACKGROUND 

Litton was first trained in inspections by Tom Gilb in 1989. His method differs from 
Fagan’s [Gilb 88], and Litton has subsequently modified Gilb's method for in-house. The 
success of our program owes much to strong executive support. Inspections are now the 
cornerstone of our peer review process. 

Over 400 software personnel have been trained in inspections, and inspections are 
now being used on four major development programs. Our software director set project 
goals to save at least 50% of integration effort by spending more effort during design and 
coding for inspections. Thus far, we appear to be achieving this goal. 

Unique properties of the Litton inspection process include no "reader" role, no 
discussion on defect category during inspection, a routing process for inspection results, no 
time limit on causal analysis and the use of a Software Engineering Process Group (SEPG) 
Peer Review Coordinator. A standard reporting form, as shown in Table 1, has been 
devised for collecting the inspection data. 

Though project management has collected some high-level inspection statistics, the 
SEPG instituted an inspection database as part of its metrics program to evaluate process 
improvement. Data from the form in Table 1 goes into the database, and is regularly entered 
at the end of each week. The database was used for this analysis, and validated against high- 
level project management data. The provision on the form for defect categories supporting 
causal analysis is a recent addition, so little data has been collected for defect category 
analysis up to this point. The following sections describe some results-to-date of our analysis 
of the inspection data. 
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Table 1 : Typical Data Sheet 


INSPECTION STATISTICS 


MODERATOR: DATE: 16 November. ;??? 

SUBJECT: DP 18. 14 CHUNK: 1 SUBJECT TYPE: Detail Design 

PRE INSPECTION MEETING DATA 


INSPECTOR 

A 


PREPARATION 
TIME (minutes) 

40 

MAJORS 

1 

MINORS 

1 

TOTAL 

ITEMS 

2 

B 



60 

0 

6 

6 

C 



60 

0 

1 _ 

1 

D 



30 

0 

5 

5 

E 

F 

TOTALS 



190 

1 

13 

li 






INSPECTIOK MEETING DATA 

Estimated SLOCs (from 

FDB > : 

N/A 




Changed Pagee/Changed 

Lines 

Inspected : 

S50 Start Time: 9:09 

— 

Total MAJORS 

Asserted 


0 

Stop Time: 

9:40 


Total MINORS 

Asserted 


-22 

Inspection 

Time (min) : 

_2i 


Total Defect* Asserted: 22 Defects Asserted Per Minute: .76 


Changed Pages/Changed Lines Inspected Per Hour: 
Hew Defect# Found During Meeting: 5 


POST INSPECTION MEETING DATA 


Total MAJORS Accepted: 2 Total Minors Accepted: 

Rework Hours: 4 Hours Working Causal Analysis Items: 

Number of Causal Analysis Items Requiring Action: Hone 
Category Totals: 1 : 2 2 : 1 3 : 0 4 : I . 5 : 4 6: 0 

9:__£ 10:_3 il:_2 12:_£ 13:.7 


IS. 

H/A 


ANALYSIS 

This analysis concerns both optimization of the inspection process, as well as 
performing a cost/benefit analysis to determine how much extra effort is used during design 
and coding for inspections and how much is saved during testing and integration. This effect 
on project effort is shown in Figure 1 from [Fagan 86]. 
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Figure 1 : Effect of Inspections on Project Effort (from [Fagan 86]) 
The following formulations are used in this analysis: 
defects found = items from preparation + new items 
inspection effort = preparation effort + meeting effort + rework effort 
defect removal effectiveness = defects found / inspection effort 
finding rate = defects / meeting time 
inspection rate = inspected pages / meeting time 


meeting effort = (meeting time) * (# personnel involved). 

Preparation effort is the total effort for all inspectors. A major defect is defined as an error 
that would lead to a trouble report during testing and integration. A new item is one found 
during the inspection meeting that was not identified by any inspectors during pre-inspection 
preparation. We decided to separate new items discovered at the inspection meeting from 
defects noted during preparation, as we have observed that certain practices increase the new 
item finding rate and wish to investigate further. 

Several types of documents are inspected: requirements (both requirements 
description and requirements analysis); design (top-level and detailed design); code; and 
change requests. Summary statistics are shown in Table 2. The total inspection effort was 
distributed as follows: preparation effort - 27%, inspection meeting effort - 39% and rework 
effort - 34%. The last column in Table 2 represents the defect removal effectiveness. As 
seen, the effectiveness decreases for later documents. 
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Table 2: Summary Statistics 


Subject Type 

Total Defects 

Total Majors 

inspection 

Effort 

LOC Major Defects/ 

# Pages Inspected Inspection Effort 

REQUIREMENT DESCRIPTION 

460 

72 

78 

179 

0 

.923 

REQUIREMENT ANALYSIS 

2165 

177 

483 

1065 

0 

.366 

HIGH LEVEL DESIGN 

2199 

188 

655 

1592 

0 

.287 

DETAILED DESIGN 

1550 

127 

610 

1387 

19007 

.208 

Subtotal 

6374 

564 

1826 

4223 

19007 

.309 

CODE 

4272 

432 

1742 

5047 

149361 

.248 

CHANGE REQUEST 

814 

27 

309 

1579 

0 

.087 

Grand total 

11460 

1023 

3877 

10849 

168368 

.264 


When the defect density for these document types are ordered by activity, the results 
show that the defects steadily decrease since the predecessor artifacts were previously 
inspected. This is shown in Figure 2, overlayed with similar results from JPL [Kelly-Sherif 
90]. The trend seems to corroborate the previous results. Code is not shown because of 
inconsistencies in reporting the document size. These results strongly support the practice of 
inspecting documents early as possible in the life cycle. 



Requirements Requirements High Level Low Level Code Change 

Description Analysis Design Design Request 


Inspection Type 


Figure 2: Defect Density per Subject Type 
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Project Effort for Inspections 

We tracked the inspection effort as a portion of the total software development effort 
over the last year. The effects of schedule pressure were seen on inspection data, much as it 
is observed for staff coding and integration efforts before a "drop dead" date. This trend is 
shown in Figure 3, where the percent of project effort dedicated to inspections is plotted. 
The monthly inspection effort profile shows extreme peaks right before two Technical 
Interface Meetings where the customer evaluates the inspected documents, and right before a 
Preliminary Design Review with the customer. For this time period of regular inspections, 
an average of 2.9% of effort was spent on inspections. Both preparation time and 
inspection time increased during the peaks, but preparation time increased much more 
severely. The relatively small increase in actual inspection time indicates that the meeting 
process remained under control, instead of moderators drastically slowing the pace to find 
more defects. These dynamic effects on effort due to schedule pressure affect the long term 
averages and short-term project behavior, and should be kept in mind when planning effort 
or evaluating project trends since process stability is affected. 



Month 


Figure 3: Percent of Project Effort for Inspections 

Based on statistics on the inspection effort and knowledge about the process, a 
bottoms-up inspection costing algorithm has been devised. It identifies effort for pre- 
inspection, inspection and post- inspection activities based on the type and length of the 
inspected document(s). The algorithm is being included in a cost model used in the 
Division. 

Return on Investment 

The following return on investment (ROI) method of tracking inspection success 
calculates the difference of testing time saved and inspection effort for each meeting [Grady 
92], [Rodriguez 91]. It uses the formulation 

ROI= (found defects) * (average effort to fix defect in test) - inspection effort 
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for each inspection meeting, using major defects only. The rationale for equating test defects 
with design defects follows from the previous definition of a major defect. At Litton, our 
historical data on the product line shows an average of 17.6 person-hours is spent to fix 
defects during testing. Using this value, the ROI for each inspection is shown in Figure 4. 
Overall, the total return from these inspections has been 14,210 person-hours of effort, with 
an average return of 63.4 person-hours per inspection. Out of 223 inspections, 139 have 
provided savings. 



Figure 4: Return on Investment per Inspection 


Statistics have been kept for several years on the number of trouble reports 
encountered during integration and the associated costs to fix them for this particular product 
line. When comparing trouble report data before and after inspections were introduced, 
there is a 76% reduction in trouble report density. This appears to be on the high end of 
reported results for defect reduction. Using the historical data on average efforts to fix 
inspection defects and trouble reports, about 573 labor-hours per KSLOC have been saved. 

Process Control 

Figures 5-7 show control charts for defect finding rate, design document inspection 
rate and code inspection rate. The bands shown on them represent the average values plus or 
minus a standard deviation for the upper and lower control limits. The overall items/minute 
for this project is apparently on the low end of the industry standard. The variances of 
inspection rates are higher relative to the variances of defect finding efficiency due to 
aforementioned dynamic schedule effects and other phenomena such as "process tweaking" 
and new personnel. 

In Figure 5, it appears that the defect finding rate seems to have come down since 
the beginning of the program. This trend of project evolution could be due to the earlier 
documents having higher defect densities per Figure 2. In Figure 6, note that there 
seems to be a relatively sudden ending to the activity near 5/93. This corresponds to the 
date when coding started in earnest, and much design documentation was completed at 
that time. 
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LOC/Hour Pages/Hour Items/Minute 


2.5 


2 



Date 

Figure 5: Defect Finding Rate Control Chart 



Date 

Figure 6: Document Inspection Rate Control Chart 



Date 

Figure 7: Code Inspection Rate Control Chart 


When analyzing the data for adherence to process control limits for inspection rate 
and item finding rate, several outlying data points were identified. Upon further 
investigation, it was seen that there was a single moderator who was not particularly well- 

7 


SEW Proceedings 


182 


S EL-93-003 



suited to the task. This moderator had been previously identified as one who rushed through 
the documents too fast, and the analysis confirmed that perception. 

Along these same lines of inquiry, we wanted to see if outlying moderators could be 
detected by looking at individual performance. Figure 8 shows the average items found per 
minute for all moderators, and they all are in the same approximate range. This depiction 
showed some disparate ranges between moderators earlier in the program, thus we feel that 
the process has stabilized among moderators over time. This provides confidence that the 
process is relatively independent of individual moderators used and shows the benefits of 
good training. 





Moderator # 

Figure 8: Moderator Finding Efficiency 

The inspection rate is an important parameter to optimize. Going too slow may waste 
time, but going too fast will miss defects. Figure 9 shows the average defect density for 
different ranges of inspection rate. Note that we have normalized the defects found by the 
document size. As seen, going faster than about 50 pages per hour seems to substantially 
decrease the defects found. The overall average is 48 pages per hour, though we are 
currently trying to slow down the rate at meetings to be closer to 30-40 pages per hour. 




A O O < 

^ to to r 

Inspection Rate (pages/hour) 


Figure 9: Effect of Inspection Rate on Defects Found 
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We also wish to know the optimal number of inspectors to maximize the defect 
removal effectiveness. Other studies have shown that 4-5 inspectors is the optimal number 
[Grady 92], [Gilb 88], and our data also supports this number. Figure 10 shows the average 
defect removal effectiveness for the number of inspectors. From our data, the optimum does 
not appear quite as clear-cut for major defects alone. 


0.14 T 



Number of Inspectors 


□ Minor defects 
■ Major defects 


Figure 10: Average Defect Removal Effectiveness vs. Number of Inspectors 

Yet another process parameter to optimize is the ratio of preparation time to 
inspection time. Grady and others [Grady 92] indicate an optimum value greater than 1.75, 
with some sites averaging about 1.5. Figure 11 shows our results. The optimum ratio 
appears to be somewhere between .5 and 2.0, with our average ratio being 1.4. 


2.5 



Preparation time/inspection time 


Figure 11: Defects Found vs. Preparation/Inspection Time 

One counter intuitive result not previously reported in the literature is a high 
correlation (.8) between the preparation time (averaged over the inspectors) and new 
items/page or new items/KSLOC found during the inspection. Instead of catching less new 
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defects during inspection after more thorough preparation to identify defects before the 
meeting, the inspectors are more familiar with the subject matter and thus able to find even 
more new items during the inspection meeting. A scatterplot of this data for all non-code 
documents is shown in Figure 12. 


1.2 


a> 

o> 

<0 

Q_ 0.8 


<o 

E 

a> 


0.6 


5 

0 > 


0.4 


0.2 


10 


— I 

14 


Preparation time/Page 


Figure 12: Effect of Preparation Time on New Items Found 

As expected, there were also high correlations between preparation time and total 
items found (pre- inspection and new items) and inspection time versus both total items and 
new items. These relationships are more stable for design documents as opposed to code 
documents, due to the reduced clarity and understanding of program code. 

Resulting Defect Density During Integration 

Inspections are expected to severely reduce the number of problems encountered 
during testing and integration activities. Though this project is not 100% complete, data 
from the first couple of builds supports this hypothesis. Figure 13 shows the resulting defect 
density during integration, as the trouble report density running average by build. The first 
10 builds were before inspections started, and the last two are for the current project within 
the same product line after inspections were mandated. Other project environmental factors 
are virtually identical except for the use of inspections. We are confident that something is 
helping to reduce the trouble report density. 

Attempts were also made to perform a t-test on individual modules to determine if 
there are significant differences in defect density during testing due to inspection. The 
metrics tracking procedures did not lend themselves to such analysis due to intractable 
mappings between design documents and implemented code functions, actual code sizes 
could not be mapped at a low level to what was being inspected, and the inability to 
distinguish new development from modified code. 

This experience was a lesson learned. In order to evaluate new techniques in the 
future for process improvement, the metrics procedures have to be restructured on the 
program, so that individual modules can be tracked throughout the lifecycle. 
Recommendations for the changes are being documented. 
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Figure 13: Defect Density During Integration 


CONCLUSIONS AND FUTURE WORK 

Though this initial major project using an inspection-based process is not complete, 
the preliminary results indicate a large return on investment. Since inspections began, 
inspectors have increased their effort and authors are producing higher quality documents, 
indicating buy-in to the new process. 

Some process stabilization occurred during the first year of practice, and the teaching 
method and the process itself has been modified based on the statistical results. Inspections 
are being used on more ongoing projects, and the results appear to be repeatable within the 
company. The process is now mandated on all new projects. 

This analysis has helped to identify areas of improvement for software metrics 
collection. This impetus will lead to revised procedures to enable more thorough analysis of 
process improvement activities. 

Analysis of inspection data will continue in order to understand and account for the 
confounding factors of inspectors and authors, to continue identifying optimal practices, to 
perform more detailed cost/benefit analysis and to investigate other related process issues. 
Analysis of variance will be performed to determine the contribution of different process 
parameters to overall defect removal effectiveness. 

With the recent enhancement to the data form for defect category information, defect 
metrics will be collected to support causal analysis activities. Additionally, a system 
dynamics model of an inspection-based process is under development, and will be calibrated 
to Litton data to assist in process improvement activities. 
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Unique Properties of Litton Inspection Process 


• No "reader" role (Fagan). 

• No discussion on defect category during inspection. 

• Routing process. 

• No time limit on causal analysis. 

• SEPG Peer Review Coordinator serves as moderator. 


Litton 

Data Systems 


Typical Data Sheet 
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Summary Statistics 


Subject Type 

Total Defects 

Total Majors 

Inspection 

Effort 

LOC Major Defects/ 

0 Pages Inspected Inspection Effort 

REQUIREMENT DESCRIPTION 

460 

72 

78 

179 

0 

.923 

REQUIREMENT ANALYSIS 

2165 

177 

483 

1065 

0 

366 

HIGH LEVEL DESIGN 

2199 

188 

655 

1592 

0 

.287 

DETAILED DESIGN 

1550 

127 

610 

1387 

19007 

.208 

Subtotal 

6374 

564 

1826 

4223 

19007 

.309 

CODE 

4272 

432 

1742 

5047 

149361 

.248 

CHANGE REQUEST 

814 

27 

309 

1579 

0 

087 

Grand total 

11460 

1023 

3877 

10849 

168368 

.264 
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Inspection Effort 



Month 
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Return on Investment 


• For each inspection, ROI = (test effort saved) - (inspection effort) 
where 

test effort saved = 

(# major defects found)* (average effort to fix defect during test) 
inspection effort = preparation effort + meeting effort + rework effort 
= total preparation effort 

+ (meeting time) * (# personnel involved in meeting) 

+ rework effort 
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Return on Investment 


total return = 14210 person-hours 

average inspection savings = 63.4 person-hours 

139/223 inspections saved time 
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Effect of Inspection Rate on Defects Found 
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Defect Removal Effectiveness vs 
Number of Inspectors 
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Moderator Finding Efficiency 
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Effect of Preparation/Inspection Time Ratio on 

Defects Found 
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Effect of Preparation and Inspection Time on 
New Items Found 
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Defect Finding Rate Control Chart 
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Document Inspection Rate Control Chart 
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Code Inspection Rate Control Chart 
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Resulting Defect Density During Integration 
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Conclusions and Future Work 


• Inspections are a worthwhile investment. 

• Peer review coordinator essential to keeping process under control. 

• Strong correlation between pre-inspection effort and new items found. 

• Inspections appear to affect downstream artifacts and eventual system integration. 

• Inspectors and authors have improved since inspections began. 

• Some stabilization observed during first year of practice. 

• Improved teaching method and changed process based on statistical results. 

• Inspection analysis has provided impetus for improved metrics tracking procedures. 

• Further analysis desired. 

* understand and account for confounding factors 

- defect category metrics and causal analysis 

- process control and optimization 

- ANOVA, other 
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